Practical Protection of Kernel Integrity for Commodity OS from Untrusted Extensions

Kernel extensions are widely used by attackers to compromise the operating system kernel. With the presence of various untrusted extensions, it remains a challenging problem to comprehensively preserve the integrity of OS kernels in a practical and generic way. In this paper, we present HUKO, a hypervisor-based integrity protection system designed to protect commodity OS kernels from untrusted extensions. In HUKO system, untrusted kernel extensions can safely run to provide desired functionalities. The behaviors of untrusted extensions, however, are confined by mandatory access control policies, which significantly limit the attacker’s ability to compromise the integrity of the kernel. To guarantee multi-aspect protection and enforcement, HUKO leverages hardware assisted paging to transparently isolate untrusted extensions from the OS kernel. Moreover, HUKO overcomes the challenge of mediation overhead by introducing a novel design named subject-aware protection state transition to eliminate unnecessary privilege transitions caused by mediating allowed accesses. Our approach is practical because it requires little change for either OS kernel or extensions, and it can inherently support multiple commodity operating systems and legacy extensions. We have implemented a prototype of HUKO based on the open source Xen hypervisor. The evaluation results show that HUKO can comprehensively protect the integrity for both Linux and Windows kernel from various kinds of malicious extensions with an acceptable performance cost.